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(54) High-capacity WDM-TDM packet switch 

(57) A self-configuring distributed packet switch(20) 
that operates in wavelength division multiplexed (WDM) 
and time division multiplexed (TDM) modes is de- 
scribed. The switch comprises a distributed channel 
switching core (26), that includes core modules (34) 
which are respectively connected by a plurality of chan- 
nels to a plurality of high-capacity packet switch edge 
modules (22,24). Each core module operates independ- 
ently to schedule paths between edge modules, and 



reconfigures the paths in response to dynamic changes 
in data traffic loads reported by the edge modules. 
Reconfiguration timing between the packet switch mod- 
ules and the channel switch core modules is performed 
to keep reconfiguration guard time minimized. The ad- 
vantage is a high-capacity, load-adaptive, self-configur- 
ing switch that can be distributed to serve a large geo- 
graphical area and can be scaled to hundreds of Tera 
bits per second to support applications that require very 
high bandwidth and a guaranteed quality of service. 
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Description 
TECHNICAL FIELD 

[0001] This invention relates generally to the field of 
data packet switching and, in particular, to a distributed 
very high-capacity switch having edge modules that op- 
erate in packet switching mode and core modules that 
operate in circuit switching mode, the core modules 
switching payload traffic between the edge modules us- 
ing wavelength division multiplexing (WDM) and time di- 
vision multiplexing (TDM). 

BACKGROUND OF THE INVENTION 

[0002] Introduction of the Internet to the general pub- 
lic and the exponential increase in its use has focused 
attention on high speed backbone networks and switch- 
es capable of delivering large volumes of data at very 
high rates. In addition to the demand for higher transfer 
rates, many service applications are being developed, 
or are contemplated, which require guaranteed grade of 
service and data delivery at guaranteed quality of serv- 
ice. To date, efforts to grow the capacity of the Internet 
have largely been focused on expanding the capacity 
and improving the performance of legacy network struc- 
tures and protocols. Many of the legacy network struc- 
tures are, however, difficult to scale into very high-ca- 
pacity networks. In addition, many legacy network pro- 
tocols do not provide grade of service or quality of serv- 
ice guarantees. 

[0003] Nonetheless, high capacity switches are 
known in the prior art. Prior art high capacity switches 
are commonly constructed as a multi-stage, usually 
three-stage, architecture in which ingress modules com- 
municate with egress modules through a switch core 
stage. The transfer of data from the ingress modules to 
the egress modules must be carefully coordinated to 
prevent contention and to maximize the throughput of 
the switch. Within the switch, the control may be distrib- 
uted or centralized. A centralized controller must receive 
traffic state information from each of the ingress mod- 
ules. Each ingress module reports the volume of waiting 
traffic destined to each of the egress modules. The cen- 
tralized controller therefore receives traffic information 
related to traffic volume from each of the ingress mod- 
ules. If. in addition, the controller is made aware of the 
class of service distinctions among the waiting traffic, 
the amount of traffic information increases proportional- 
ly. Increasing the amount of traffic information increases 
the number of control variables and results in increasing 
the computational effort required to allocate the ingress/ 
egress capacity and to schedule its usage. Consequent- 
ly, it is desirable to keep the centralized controller una- 
ware of the class of service distinctions while providing 
a means of taking the class of service distinctions into 
account during the ingress/egress transfer control proc- 
ess. 



[0004] This is accomplished in a rate-controlled multi- 
class high-capacity packet switch described in Appli- 
cant's copending European Patent Application No. 
00300700.2 which was filed on January 31, 2000. Al- 

5 though the switch described in that patent application is 
adapted to switch variable sized packets at very high 
speeds while providing grade-of-service and qualiiy-of- 
service control, there still exists a need for a distributed 
. switch that can form the core of a powerful high-capac- 

10 ity, high-performance network that is adapted to provide 
wide geographical coverage with end-to-end capacity 
that scales to hundreds of Tera bits per second (Tbs), 
while providing grade of service and quality of service 
controls. 

15 [0005] A further challenge in providing a powerful 
high-capacity, high-performance switch with wide geo- 
graphical coverage is maintaining network efficiency in 
the face of constantly fluctuating traffic volumes. In re- 
sponse to this challenge, the Applicant also invented a 
20 self-configuring data switch comprising a number of 
electronic switching modules interconnected by a sin- 
gle-stage channel switch that includes a number parallel 
space switches, each having input ports and output 
ports. This switch architecture is described in Appli- 
es cants copending European Patent Application entitled 
SELF-CONFIGURING DISTRIBUTED SWITCH which 
was filed on April 5, 2000 and assigned Application No. 
00302888.3. Each of the electronic modules is capable 
of switching variable-sized packets and is connected to 
so the set of parallel space switches by a number of optical 
channels, each of the optical channels being a single 
wavelength in a multiple wavelength fiber link. The 
channel switching core permits any two modules to be 
connected by an integer number of channels. In order 
35 to enable the switching of traffic at arbitrary transfer 
rates, the inter-module connection pattern is changed 
in response to fluctuations in data traffic load. However, 
given the speed of optical switching equipment and the 
granularity of the channels, it is not always possible to 
40 adaptively modify the paths between modules to accom- 
modate all data traffic variations . Consequently, it some- 
times proves uneconomical to establish under-utilized 
paths for node pairs with low traffic volumes. To over- 
come this difficulty, a portion of the data traffic flowing 
^5 between a source module and a sink module is switched 
through one or more intermediate nodes. Thus, in effect, 
the switch functions as a hybrid of a channel switch and 
linked buffer data switch, benefiting from the elastic path 
capacity of the channel switch. 
so [0006] A concentration of switching capacity in one lo- 
cation is, however, undesirable for reasons of security 
and economics. The self-configuring distributed switch 
with a high capacity optical core described in Applicant's 
co-pending Patent Application is limited in capacity and 
55 limited to switching entire channels. Consequently, it is 
desirable to provide a high-capacity switch with a dis- 
tributed core. Such a core has the advantages of being 
less vulnerable to destruction in the event of a natural 
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disaster, for example. It is also more economical be- 
cause strategic placement of distributed core modules 
reduces link lengths, requires less concentration of in- 
frastructure and provides shorter paths for localized da- 
ta traffic. 

[0007] There therefore exists a need for a very high- 
capacity packet switch with a distributed core that is 
adapted to provide grade of service and quality of serv- 
ice guarantees. There also exists a need for a very high- 
capacity packet switch that provides intra-switch data 
paths of a finer granularity to reduce or eliminate a re- 
quirement for tandem switching. 

SUMMARY OF THE INVENTION 

[0008] According to the invention, there is provided a 
high capacity packet switch comprising a plurality of 
core modules, each of the core modules operating in a 
circuit switching mode, a plurality of edge modules con- 
nected to subtending packet sources and subtending 
packet sinks, each of the edge modules operating in a 
data packet switching mode. CHARACTERIZED by: 

the core modules switch the data packets between 
the edge modules using wavelength division multiplex- 
ing (WDM) and time division multiplexing (TDM). 
[0009] The high-capacity packet switch of the inven- 
tion switches data packets between the edge modules 
using wavelength division multiplexing (WDM) and time 
division multiplexing (TDM). The switch has a distribut- 
ed core that is adapted to provide guaranteed grade of 
service and quality of service. The switch of the inven- 
- -tion can support intra-switch data paths with a granular- 
ity that reduces or eliminates a requirement for tandem 
switching. 

[0010] The plurality of core modules can be geo- 
graphically distributed and can operate independently. 
They may also periodically reconfigure to adapt to fluc- 
tuations in the network traffic. The core modules may 
coordinate reconfiguration with the edge modules in or- 
der to minimize a reconfiguration guard time. 
[0011] Each of the core modules is preferably a space 
switch. Any of the well known textbook designs for a 
space switch can be used. However, the preferred 
space switch is an electronic single-stage rotator switch , 
because of its simple architecture, ease of control and 
scalability. The core modules preferably have neither in- 
put nor output buffers. A one of the edge modules is pref- 
erably co-located with each core module and serves as 
a controller for the core module. 
[0012] Each of the edge modules has a plurality of in- 
gress ports and a plurality of egress ports. Each of the 
ingress ports has an associated ingress queue. An in- 
gress scheduler sorts packets arriving in the ingress 
queues from the subtending packet sources, the sort be- 
ing by destination edge module from which the respec- 
tive packets are to egress from the high capacity packet 
switch for delivery to the subtending packet sinks. The 
ingress scheduler periodically determines a number of 
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packets waiting in the ingress queues for each other re- 
spective edge module, and sends a payload traffic allo- 
cation vector to each of the controllers of the core mod- 
ules. The traffic allocation vector sent to a given control- 

5 ler relates only to a group of channels that connect the 
edge module to the given core module. 
[0013] Each edge module also maintains a vector of 
pointers to the sorted payload packets, the vector of 
pointers being arranged in egress edge module order. 

10 A scheduling matrix for each slot in a time frame and 
each egress edge module is associated with the vector 
of pointers and determines a data transfer schedule for 
the ingress edge module. 

[0014] Each ingress edge module also maintains .an 

15 array of reconfiguration timing circuits, a one of the 
reconfiguration timing circuits being associated with 
each of the core modules. The reconfiguration timing cir- 
cuits are respectively synchronized with time clocks in 
the respective edge modules that serve as controllers 

20 for the core modules . The reconfiguration timing circuits 
enable reconfiguration of channel switching in the core 
modules using a short guard time. 
[001 5] Each core module preferably comprises a plu- 
rality of rotator switches. Each rotator switch preferably 

25 accommodates a number of input channels equal to the 
number of edge modules, as well as a number of output 
channels equal to the number of edge modules. In a 
folded edge module configuration, each edge module 
preferably has one channel connected to an input port 

30 and one channel connected to an output port of each 
rotator* switch, in an unfolded edge module configura- 
tion, each edge module is either an ingress module or 
an egress module. The ingress and egress modules are 
preferably arranged in co-located pairs. In the unfolded 

35 configuration, each ingress edge module preferably has 
one channel connected to an input port of each rotator 
switch. Each egress module likewise preferably has one 
channel connected to an output port of each rotator 
switch. 

40 [0016] The invention also provides a method of 
switching payload data packets through a distributed 
data packet switch, comprising steps of receiving a pay- 
load data packet from a subtending source at an ingress 
edge module of the distributed data packets switch, de- 

45 termining an identity of an egress edge module from 
which the data packet should egress from the distributed 
data packet switch, arranging the data packet in a sorted 
order with other data packets received so that the data 
packets are arranged in a sorted order corresponding 

so to the identity of the edge module from which the data 
packet should egress from the distributed data packet 
switch, transferring the sorted data packets in fixed- 
length data blocks to a core module of the distributed 
data packet switch, switching the fixed-length data 

55 blocks through the core module to the egress module, 
reconstructing the data packet at the egress edge mod- 
ule, and transferring the data packet from the egress 
module to a subtending sink CHARACTERIZED by: 
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using wave division multiplexing (WDM) and time 
division multiplexing (TDM) to provide intra-switch data 
paths of a fine granularity to reduce an requirement for 
tandem switching in the distributed data packet switch. 
[0017] An ingress scheduler may periodically deter- 5 
mine a number ol packets wailing in ingress queues for 
each respective egress edge module and sends a ca- 
pacity request vector to each of the controllers of the 
core modules, the capacity request vector sent to a giv- 
en controller relating only to a group of channels that 10 
connect the edge module to the given core module. 
[0018] Each ingress edge module may maintain a 
vector of pointers to the packets sorted by egress edge 
module and a scheduling matrix that provides a port 
number for each slot in which a data block can be trans- « 
ferred, the scheduling matrix being arranged in the 
same egress edge module order so that the scheduling 
matrix and the pointers are logically aligned; and, when 
a non-blank entry in the scheduling matrix indicates an 
egress port through which a data block can be trans- 20 
ferred, a corresponding pointer in the vector of pointers 
is used to locate a starting point for the data block in the 
packets waiting in the ingress queues. 
[001 9] Preferably, an array stores pointers to packets 
sorted by egress edge module, and the pointers are dy- 25 
namically updated each time a data block is transferred 
from the ingress queues to an egress channel so that 
reconfigurations of the core modules to the data can be 
accomplished without queue flushing. Two scheduling 
matrices, one in current use and one in update mode, 30 
are maintained at each ingress module. Each time a 
core reconfiguration occurs, a scheduling matrix in use 
is swapped for a current scheduling matrix. An unused 
copy of the scheduling matrix is available for update af- 
ter the core reconfiguration. Rows in the matrix are ex- 35 
ecuted sequentially, one per time slot, until a next core 
module reconfiguration occurs. After core module 
reconfiguration, processing continues at a next time 
slot. 

40 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0020] The invention will now be explained by way of 
example only, and with reference to the following draw- 
ings, in which: 45 

FIG. 1 is a schematic diagram of a high capacity 
WDM-TDM packet switch in accordance with the in- 
vention having a centralized core; 
FIG. 2 is a schematic diagram of the high capacity so 
WDM-TDM packet switch shown in FIG. 1 wherein 
the space switches in the core are single-stage ro- 
tator switches; 

FIG. 3 is a schematic diagram of a high capacity 
WDM-TDM packet switch in accordance with the in- 55 
vention with a distributed core; 
FIG. 4 is a schematic diagram of a high capacity 
WDM-TDM packet switch in accordance with the in- 



vention showing an exemplary distribution of the 

core modules and edge modules; 

FIG. 5 is a schematic diagram of a data structure 

used in each edge module to facilitate a process of 

computing capacity request vectors in the edge 

modules; 

FIG. 6 is a schematic diagram of a table used by an 
ingress edge module to determine a preferred core 
module for a connection to an egress module; 
FIG. 7 is a schematic diagram of data structures 
used in each core module for capacity scheduling 
using capacity request vectors received from the 
edge modules; 

FIG. 8 is a schematic diagram illustrating space 
switch occupancy in a four core-module distributed 
switch in which a matching method employing a 
packing-search discipline is used; and 
FIG. 9 is a schematic diagram of data structures 
used to control the transfer of data blocks from an 
ingress module to core modules of a high capacity 
WDM-TDM packet switch in accordance with the in- 
vention. 

[0021 ] It should be noted that throughout the append- 
ed drawings, like features are identified by like reference 
numerals. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

[0022] FIG. 1 is a schematic diagram of a high-capac- 
ity WDM-TDM packet switch in accordance with the in- 
vention , generally indicated by reference 20. The packet 
switch 20 includes a plurality of edge modules 22, 24 
shown for clarity of illustration in an "unfolded" configu- 
ration. In the unfolded configuration shown in FIG. 1, 
ingress edge modules 22 and egress edge modules 24 
are separate switching modules constructed, lor exam- 
ple, as described in Applicant's copending Patent Appli- 
cation Serial No. 00300700.2 which was filed January 
31. 2000 and entitled RATE-CONTROLLED MULTI- 
CLASS HIGH-CAPACITY PACKET SWITCH, which de- 
scribes the architecture and control of a high-capacity 
packet switch based on a rotator architecture. In a folded 
switch configuration, the ingress edge modules 22 and 
the egress edge modules 24 are combined into integrat- 
ed switch modules of one ingress module and one 
egress module each, each integrated module having as 
many data ports as a sum of the data ports of the ingress 
edge module 22 and the egress edge module 24. 
[0023] Located between the edge module pairs 22, 24 
are a plurality of space switches 26 which serve as cen- 
tralized core modules for the WDM-TDM packet switch 
20. For the sake of scalability and switching speed, the 
space switches 26 are preferably electronic space 
switches, although optical space switches could be 
used and may become preferred when optical switching 
speeds improve. The space switches 26 are arranged 
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in parallel and, as will be described below, are preferably 
distributed in collocated groups. The number of edge 
modules 22, 24 and the number of space switches 26 
included in the WDM-TDM packet switch 20 is depend- 
ent on the switching capacity required, in the example 5 
shown in FIG. 1 , there are 256 (numbered as 0-255) in- 
gress edge modules 22 and 256 egress edge modules 
24. Each edge module 22 has egress ports to support 
128 channels. In a typical WDM multiplexer, 16 wave- 
lengths are supported on a link. Each wavelength con- io 
stitutes a channel. Consequently, the 128 channels can 
be supported by eight optical fibers if cross-connectors 
are used, as will be explained below with reference to 
FIG. 3. 

[0024] In order to ensure that any edge module 22 is 
enabled to send all of its payload traffic to any edge mod- 
ule 24, if so desired, each space switch 26 preferably 
supports one input channel for each module 22 and one 
output channel for each module 24. Therefore, in the 
example shown in FIG. 1 , each space switch preferably 
supports 256 input channels and 256 output channels. 
The number of space switches 26 is preferably equal to 
the number of channels supported by each edge module 
22, 24. In the example shown in FIG. 1 , there are pref- 
erably 128 space switches 26, the number of space 
switches being equal to the number of channels from 
each ingress module 22. 

[0025] FIG. 2 is a schematic diagram of a preferred 
embodiment of the WDM-TDM packet switch shown in 
FIG. 1. In accordance with a preferred embodiment, 
each of the space switches 26 is a single-stage rotator- 
based switch. In the rotator-based switch architecture, 
a space switch core is implemented as a bank of inde- 
pendent memories 28 that connect to the edge modules 
22 of the switch through an ingress rotator 30. Traffic is 
transferred to the edge modules 24 of the switch 20 
through an egress rotator 32. The two rotators 30, 32 
are synchronized. A detailed description of the rotator 
switch architecture is provided in European Patent No. 
EP857383A1 that issued to Applicant on August 12, 
1998, and provides a detailed description of the princi- 
ples of operation of that switch architecture. In other re- 
spects, the switch architecture shown in FIG. 2 is iden- 
tical to that shown in FIG. 1 . 

[0026] In the rotator switches 26, each bank of inde- 
pendent memories 28 is divided into a plurality of mem- 
ory sections arranged in series. Each memory is prefer- 
ably arranged in columns that are 128 bytes wide. Each 
memory is divided into a number of partitions, the 
number of partitions being equal to the number of egress 
edge module 24. The size of the memory portion gov- 
erns a size of data block switched by the channel switch- 
ing core. The size of the data block is a matter of design 
choice, but is preferably about 4-16 kilobits. 

Partitioning the Core 

[0027] The channel switching core is preferably par- 



titioned into core modules and distributed for two prin- 
cipal reasons: economics and security. FIG. 3 is a sche- 
matic diagram of a preferred embodiment of a distribut- 
ed WDM-TDM packet switch in accordance with the in- 
vention. A plurality of core modules 34 are geographi- 
cally distributed. A plurality of cross-connectors 36, 
which may be, for example, very slow optical switches, 
connect a plurality of ingress and egress edge modules 
22, 24 to the core modules 34. The cross-connectors 36 
serve as multiplexers and thereby reduce the number 
of physical links required to connect each ingress and 
egress edge module 22, 24 to each core module 34 . The 
core modules 34 preferably include an equal number of 
rotator switches. A WDM-TDM packet switch 20 of a 
size shown in FIGs. 1 and 2, with eight core modules 
34, includes 1 6 rotator switches 28 in each core module 
34 when geographically distributed as shown in FIG. 3 
(8x16= 128). If the ingress and egress edge modules 
22, 24 are grouped in clusters of eight per cross-con- 
nector 36, then 64 cross-connectors are required to con- 
nect the ingress and egress edge modules 22, 24 to the 
core modules 34. The clustering of the ingress and 
egress edge modules 22, 24 and the number of cross- 
connectors 36 used in any given installation is depend- 
ent on network design principles well understood in the 
art and does not require further explanation. In any dis- 
tributed deployment of the WDM-TDM packet switches, 
it is preferred that each ingress and egress edge module 
22, 24 be connected to each space switch 26 of each 
core module 34 by at least one channel. The switch may 
be partitioned and distributed as desired with the excep- 
tion that one ingress and egress edge module 22 is pref- 
erably collocated with each core module 34 and serves 
or hosts a controller, as a controller for the core module, 
as will be explained below in more detail. 
[0028] FIG. 4 shows an exemplary distribution of a 
WDM-TDM packet switch 20 in accordance with the in- 
vention, to illustrate a hypothetical geographical distri- 
bution of the switch. Cross-connectors 36 and optical 
40 links 38 are not shown in FIG. 4 for the sake of clarity. 
In this example, 16 ingress and egress edge modules 
22, 24 numbered 0-1 5 and four core modules 34 num- 
bered 0-3 are potentially distributed over a large geo- 
graphical area. As explained above, an ingress edge 
45 module 22 is collocated with each core module 34. In 
this example, ingress edge modules 0-3 are collated 
with corresponding core modules 0-3. Because the 
space switches 26 are rote devices with substantially no 
computational capacity, they require controllers to per- 
so form scheduling allocations and other functions which 
are described below in more detail. The ingress edge 
modules 22 include high-speed processors which are 
capable of performing control functions, or hosting con- 
trol functions, for the core modules 34. Consequently, 
55 an ingress edge module 22 is preferably collocated with 
each core module 34. The processor of the ingress edge 
module 22 need not, however, perform the control func- 
tions of the core module 34. Rather, it may host, at one 
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of its ports, a processor to perform the control functions 
of the core module 34. The collocation is also important 
to enable time coordination in the distributed WDM-TDM 
packet switch 20, as explained below. 

5 

Time Coordination in the Distributed WDM-TDM 
Packet Switch 

[0029] Time coordination is required between ingress 
edge modules 22 and core modules 34 if the WDM-TDM 10 
packet switch 20 geographically distributed. Time coor- 
dination is necessary because of propagation delays 
between ingress edge modules 22 and the core mod- 
ules 34. Time coordination is accomplished using a 
method described in Applicant's above -referenced co- '5 
pending patent application filed April 4, 1999. In accord- 
ance with that method, time coordination is accom- 
plished using an exchange of timing packets between 
the ingress edge modules 22 and the respective edge 
module controller for core modules 34. At predeter- 20 
mined intervals, each ingress edge module 22 is pro- 
grammed to send a timing packet to the ingress edge 
module 22 that serves as a controller for the associated 
core module 34. For example, ingress edge module 9 
(FIG. 4) at a predetermined interval sends a timing pack- 25 
et to ingress edge module 3 associated with core mod- 
ule 3. On receipt of the timing packet, the ingress edge 
module 3, which serves as a controller for the core mod- 
ule 3, stamps the packet with a time stamp that indicates 
its local time. At some convenient time prior to the next 30 
predetermined interval, the time stamped packet is re- 
- turned to the edge module 9. The edge module 9, and 
each of the other ingress edge modules 0-15, maintains 
an array of M reconfiguration liming circuils where M 
equals the number of core modules 34. The core mod- 35 
ules 34 operate independently and reconfigure inde- 
pendently, as will be described below in more detail. 
Consequently, each ingress edge module 22 must main- 
tain a separate reconfiguration timing circuit coordinat- 
ed with a local time of an ingress edge module 22 col- *o 
located with each core module 34. Without timing coor- 
dination, guard times for reconfiguration of the core 
modules 34 would have to be too long due to the prop- 
agation delays between the geographically distributed 
ingress edge modules 22 and the core modules 34. 45 
[0030] For example, in the configuration of the WDM- 
TDM packet switch 20 shown in FIG. 4, each ingress 
edge module 22 must maintain an array of four recon- 
figuration timing circuits respectively coordinated with 
the local times of ingress edge modules 0-3 collocated so 
with the respective core modules 34. As explained 
above, in order to maintain time coordination, the in- 
gress edge module 9. at regular predetermined inter- 
vals, sends a timing packet to the ingress edge module 
0; The timing packet is sent over a communications time ss 
slot and received on an ingress port of the ingress edge 
module 0 dedicated to management functions. The in- 
gress port, on receipt of the timing packet, time stamps 



the packet with the time from its local time (timing circuit 
0) and queues the timing packet for return to the edge 
module 9. At some convenient later time before the start 
of the next timing interval, the timing packet is returned 
to the ingress edge module 9. On receipt of the timing 
packet at ingress edge module 9, the ingress edge mod- 
ule 9 uses the time at which the packet was received at 
ingress edge module 0 (time stamp) in order to coordi- 
nate its reconfiguration timing circuit 0 with the local time 
of ingress edge module 0. Several methods for timing 
coordination are explained in detail in Applicant's co- 
pending Patent Application Serial No. 09/286.431 filed 
April 6, 1999. 

Packet Transfer Through the WDM-TDM Packet 
Switch 

[0031 ] Ingress and egress edge modules 22, 24 of the 
WDM-TDM packet switch 20 operate in packet switch- 
ing mode. The edge modules 22, 24 are adapted to 
switch variable sized packets and transfer the packets 
to subtending sinks in the format in which the packets 
were received. Switching in the core modules 34 is ac- 
complished in circuit switching mode. The core modules 
34 are completely unaware of the content switched and 
simply switch data blocks. In order to improve resource 
allocation granularity, the WDM-TDM packet switch 20 
switches in both wave division multiplexing (WDM) and 
time division multiplexing (TDM) modes. Each link 38 
(FIG. 3) interconnecting the switched edge modules 22, 
24 and the core modules 34 is preferably an optical link 
carrying WDM data on a number of channels, each 
channel being one wave length in the WDM optical link 
38. Each channel is further divided into a plurality of dis- 
crete time slots, hereinafter referred to simply as "slots". 
The number of slots in a channel is a matter of design 
choice. In a preferred embodiment, each channel is di- 
vided into 16 time slots. Consequently, the smallest as- 
signable increment of bandwidth is 1/1 6 th of the channel 
capacity. For a 10 gigabit per second (10 Gb/s) channel, 
the smallest assignable capacity allocation is about 625 
megabits per second (625 Mb/s). Connections requiring 
less capacity are aggregated by class-of-service and 
quality-of-service in a manner well known in the art. 
Connections requiring more capacity are allocated mul- 
tiple slots, as required. 

Admission Control 

[0032] The capacity requirement for each connection 
established through the WDM-TDM packet switch 20 is 
determined either by a specification received from a 
subtending source or, preferably, by automated traffic 
measuring mechanisms based on traffic monitoring and 
inference. If automated measurement is used, the ca- 
pacity requirements are expressed as a number of slots 
for high bandwidth connections. For aggregated traffic, 
the capacity requirements are measured for a class of 
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service. Regardless of the method used to estimate the 
capacity requirements, it is the responsibility of the in- 
gress edge modules 22 to quantify the traffic require- 
ments for its traffic load. It is also the responsibility of 
the ingress edge modules 22 to select a route for each 
admission request. Route selection is accomplished us- 
ing connection tables provided by a Network Manage- 
ment System (NMS) (not illustrated) which provides a 
table of preferred connecting core modules between 
each ingress and egress edge module. 
[0033] Admission control may be implemented in a 
number of ways that are well known in the art, but the 
concentration of responsibility is at the edge and any 
ingress edge module 22 receiving an admission request 
first determines whether free capacity is available on 
any of the preferred routes through a core module de- 
fined in its connection table prior to acceptance. 

Scheduling at the Edge 

[0034] At any given time, each ingress edge module 
22 has an allocated capacity to each egress edge mod- 
ule 24 expressed as a number of slots. The number of 
allocated slots depends on a capacity allocation, which 
may be 0 for certain ingress/egress module pairs. The 
allocated capacities may be modified at regular recon- 
figuration intervals which are independently controlled 
by the controllers of the distributed core modules 34. An 
ingress edge module 22 accepts new connections 
based on its current capacity allocation to each egress 
edge module 24. The controller of each ingress edge 
module 22 also monitors its ingress queues, which are 
sorted by egress edge module, as described above, to 
determine whether a change in capacity allocation is 
warranted. It is the responsibility of each ingress edge 
module 22 to determine when resources should be al- 
located and when resources should be released. How- 
ever, it is the controllers at the core modules 34 that de- 
termine whether a bandwidth allocation request can be 
granted. Bandwidth release requests are always ac- 
cepted by the controllers of the core modules 34. The 
re-allocation of bandwidth and the reconfiguration of the 
core modules 34 is explained below in more detail. 
[0035] Each ingress edge module 22 determines its 
bandwidth capacity requirements and communicates 
those requirements to the controllers of the respective 
core modules 34. On receipt of a capacity requirement, 
a controller of a core module 34 attempts to satisfy the 
requirement using a rate matching process. The con- 
troller of the core modules 34 compute a scheduling ma- 
trix based on the capacity requirements reported by 
each ingress edge module 22, as will be explained be- 
low, and returns relevant portions of the scheduling ma- 
trix to each ingress edge module 24 prior to a reconfig- 
uration of the core module 34. At reconfiguration, three 
functions are implemented. Those functions are: a) re- 
leases, which return unused resources to a resource 
pool; b) bandwidth capacity increases which allocate 



new bandwidth to ingress edge modules 22 requiring it; 
and c) new bandwidth capacity allocations, in which an 
allocation for an ingress edge module 22 is increased 
from 0. 

5 

Capacity Scheduling 

[0036] As described above, the ingress edge modules 
22 are responsible for managing their bandwidth capac- 

10 ity requirements. Consequently, each edge module 
computes a capacity requirement vector at predeter- 
mined intervals such that the capacity requirement is re- 
ported to each core module 34 at least once between 
each core reconfiguration. FIG. 5 illustrates the compu- 

i5 tation of the capacity requirement vector. As shown, an 
ingress edge module 22 constructs a matrix of x rows 
and y columns, where x is the number of ingress ports 
and y is the number of egress modules in the switch con- 
figuration. In the example shown, the number of chan- 
ge nels is 128 and number of ingress edge modules 22 is 
255. A number representative of an actual count of 
packets in the egress buffers, or a number resulting from 
a traffic prediction algorithm, is inserted in each cell of 
the matrix shown in FIG. 5. A traffic allocation require- 

25 ment sum 40 provides a summation for each egress 
edge module 24 of the total capacity requirement. The 
total capacity requirement is then subdivided into M ca- 
pacity requirement vectors, where M is the number of 
core modules 34 and the respective capacity require- 

30 ment vectors are distributed to the respective core mod- 
ules to communicate the capacity requirements. A zero 
in a capacity requirement vector indicates that any 
bandwidth capacity previously allocated to the ingress 
core module 22 is to be released. 

35 [0037] In order for an ingress edge module 22 to in- 
telligently request a bandwidth capacity increase, it 
must follow a governing procedure. As described above, 
each ingress edge module 22 is provided with a table of 
preferred connections to each egress edge module 24. 

40 FIG. 6 shows how the table of preferred connections 
through the switch is used in the bandwidth allocation 
request process. A preferred connection table 42 is pro- 
vided to edge module 7 in the network shown in FIG. 4. 
The preferred connection table 42 provides the edge 

45 module 7 with the core modules through which connec- 
tions can be made with egress edge modules, the core 
module numbers being listed in a preferred order from 
top to bottom. Each entry 44 in the preferred connection 
table 42 is a core module identification number. There- 

50 fore, if ingress edge module 7 needs to send packets to 
egress edge module 0, the preferred core module for 
the connection is core module 0. The other core mod- 
ules that may be used for the connection are, in de- 
scending order of preference, 3, 1 and 2. Likewise, if 
55 edge module 7 needs to send packets to edge module 
1 5, the preferred core module is core module 3, and the 
alternate core modules, in descending preference, are 
2, 0 and 1. 
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[0038] As shown in FIG. 6, the preferred connection 
table 42 is used in each edge module to facilitate the 
process of requesting bandwidth capacity allocations 
from the respective core modules 34! The array 40 of 
the traffic allocation summary computed as described 
above, has 16 entries, one entry for traffic destined lo 
each egress edge module. The array is matched with 
the preferred connection table 42, which has 16 col- 
umns and four rows, as explained above. The array 40 
indicates the number of slots required to accommodate 
traffic from the edge module 7 to the 1 5 other edge mod- 
ules in the network shown in FIG. 4. These data struc- 
tures are used to construct the capacity request vectors 
described above, which are sent to the respective core 
modules 34. As will be explained below in more detail, 
reconfiguration of the core modules is preferably stag- 
gered so two core modules do not reconfigure at the 
same time. Consequently, there is a staggered recon- 
figuration of the core modules 34. For each capacity re- 
quest vector sent by an ingress edge module 22, a first 
set of capacity request vectors is preferably constructed 
using the preferred connections listed in row 1 of the 
preferred connection table 42. If a capacity request de- 
nial is received back from a core module, an updated 
capacity request vector is sent to a second choice mod- 
ule. In planning capacity allocations prior to reconfigu- 
ration, a core module preferably uses the last received 
allocation request vector until processing has advanced 
to a point that any new capacity request vectors are ig- 
nored. Consequently, for example, the capacity request 
vector sent to core module 0 would request five slots for 
—a connection to egress edge module 0, seven slots for 
a connection to edge module 11 , seven slots for a con- 
nection to edge module 13, and ten slots for a connec- 
tion to edge module 14. If core module 0 denied any one 
of the capacity requests, an updated capacity request 
vector would be sent to the next preferred core module 
shown in the preferred connection table 42. 
[0039] FIG. 7 illustrates a scheduling function per- 
formed by each of the controllers for the respective core 
modules 34. Each controller for the respective core 
modules 34 receives capacity request vectors from the 
ingress edge modules 22. The capacity request vectors 
received from each ingress edge module 22 is ex- 
pressed in terms of the number of slots that each ingress 
edge module requires to accommodate its traffic 
switched through the given core module 34. The con- 
troller of each core module 34 assembles the capacity 
request vectors in a traffic allocation request matrix 44 
which includes N rows and N columns where N equals 
the number of ingress edge modules. In the example 
network shown in FIG. 4, the traffic allocation request 
matrix 44 constructed by the controller of each core 
module 34 would be a 1 6 x 16 matrix (256 x 256 matrix 
for the network shown in FIG. 3). 
[0040] The traffic allocation request matrix 44 is nor- 
mally a sparse matrix with a majority of null entries. The 
controller for the core module attempts to schedule the 



capacity requested by each ingress edge module 22 us- 
ing data structures generally indicated by references 46 
and 48. Each of the data structures 46, 48 is a three- 
dimensional matrix having a first space dimension s, 
5 which represents the respective space switches asso- 
ciated with the core module 34; a second space dimen- 
sion p, which represents the input channels; and a time 
dimension /, which represents the slots in a slotted 
frame. Thus, an entry in data structure 46 is represented 
io as {$.p,t}. The second dimension p may represent an 
input channel, if associated with the data structure 46, 
or an output channel if associated with the data structure 
48. If the number of slots Tper frame is 16, for example, 
then in the configuration of FIG. 1, which shows a cen- 
ts tralized core, the size the three-dimensional structure 
46 is 256 x 128 x 16. In the distributed core shown in 
FIG. 3, each core module uses a three-dimensional 
structure 46 of size 256 x 1 6 x 1 6. 
[0041] When the connections through a core module 
20 34 are reconfigured, the core controller may either re- 
schedule the entire capacity of the respective core mod- 
ule 34 or schedule bandwidth capacity changes by sim- 
ply perturbating a current schedule. If the entire band- 
width capacity of the core module is reconfigured, each 
25 ingress edge module 22 must communicate a complete 
capacity request vector to the core module while, in the 
latter case, each ingress edge module 22 need only re- 
port capacity request changes, whether positive or neg- 
ative, to a respective core controller. A negative change 
30 represents capacity release while a positive change in- 
dicates a request for additional bandwidth capacity. The 
incremental change method reduces the number of 
steps required to prepare for reconfiguration. However, 
the incremental change method potentially risks alloca- 
35 tion efficiency, because a core module that fails to ac- 
commodate a bandwidth capacity request may force the 
requesting ingress edge module to seek incremental ca- 
pacity allocation from another, less than optimal, con- 
nection through another core module that may lead to 
40 a longer path to a destination. 

[0042] The capacity scheduling done by the controller 
for a core module 44 can be implemented by simply 
processing the non-zero entries in the traffic allocation 
request matrix 44 one at a time. A non-zero entry 50 in 
45 the traffic allocation request matrix 44 represents a 
number of required slots for a respective edge module 
pair. A three dimensional data structure 46 indicates free 
input slots at core modules, and data structure 48 shows 
the free slots at output ports of the core module 34. The 
so three dimensional data structures 46, 48, initialized with 
null entries, are then examined to determine if there are 
sufficient matching slots to satisfy the capacity request. 
Each cell 51 in each data structures 46, 48 represents 
one slot. A slot in structure 46 and a slot in structure 48 
55 are matching slots if each is unassigned and if both have 
the same first space dimensions (s) and time dimension 
(f). Thus, slot {sj,t} in data structure 46 and slot {s,k,t} 
in data structure 48 are matching if both are free, with 
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respect to the values of y'and k. 
[0043] A bandwidth capacity request is rejected by a 
core module if sufficient matching slots cannot be found. 
In order to reduce the incidence of mismatch, the match- 
ing process should always start from a selected space 
switch at a selected lime slot and follow the same search 
path for each capacity request. For example, the match- 
ing process may start from space switch 0 at slot 0 and 
then proceed by incrementing the slot from 0 to S, where 
S is the number of time slots per channel. It then con- 
tinues to the next time port plane 53 until the 1 6 planes 
(in this example) are exhausted or the capacity is suc- 
cessfully allocated, whichever takes place first. The re- 
sult produced by this packing search, which is well 
known in the art, is an occupancy pattern shown in FIG. 
8. 

[0044] FIG. 8 shows a typical space switch occupancy 
for each of the core modules 34. Each core module 34 
includes four space switches in this example. Observing 
any of the core modules, the occupancy of the space 
switch at which the matching search always starts is 
higher than the occupancy of the second space switch 
in the search path, etc. This decreasing occupancy pat- 
tern is known to provide a superior matching perform- 
ance over methods that tend to equalize the occupancy, 
such as a round-robin or random search. 

Packet Transfer from the Edge Modules to the Core 

[0045] As a result of the scheduling, process de- 
scribed above, each core module, prior to reconfigura- 
tion, returns to each ingress edge module 22 a schedule 
vector which is used to populate a schedule matrix 54 
partially shown in FIG. 9. The schedule matrix 54 is a 
matrix containing Srows (where S= 16 in this example) 
and N columns where N equals the number of ingress 
edge modules 22. The 16 rows, only four of which are 
illustrated, represent the 16 slots in a frame. The non- 
blank entries 56 in the schedule matrix represent chan- 
nel numbers of the egress channels of an egress edge 
module 22. The edge module is enabled to transfer one 
data block to a core module 34 for each valid entry in 
the schedule matrix 54. For example, in row 1 of the ma- 
trix 54 shown in FIG. 8, the ingress edge module 22 can 
transfer a data block through ports 16 to egress edge 
module 14. In time slot 2, the edge module can transfer 
one data block through channel 97 to edge module 1. 
and one data block through channel 22 to edge module 
14. The ingress edge module 22 has no knowledge of 
the core module to which the data block is to be trans- 
ferred and requires none. 

[0046] The size of a data block is a matter of design 
choice, but in the rotator-based core modules, the size 
of a data block is related to the structure of middle mem- 
ories 28 (FIG. 2). In general, a data block is preferably 
4 kilobits (Kb) to about 16 Kb. In order for data blocks 
to be transferred from the ingress queues to the appro- 
priate egress channel, an array 58 stores pointers to 
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packets sorted by egress edge module. The pointers 58 
are dynamically updated each time a data block is trans- 
ferred from the ingress queues to an egress channel. 
Consequently, reconfigurations of the core modules 30 

5 can be accomplished without queue flushing. 

[0047] In actual implementations, it is preferable to 
maintain two matrices 54, one in current use and one in 
update mode. Each time a core reconfiguration takes 
place, the matrix in use is swapped for a current matrix. 

io The unused copy of the matrix is available for update. 
Rows in the matrix are executed sequentially one per 
slot until a next core module reconfiguration occurs. Af- 
ter core module reconfiguration, processing continues 
at the next slot. 

15 [0048] The invention thereby provides a very high- 
speed packet switch capable of wide geographical dis- 
tribution and edge-to-edge total switching capacity that 
is scalable to about 320 Tera bits per second (Tbs) using 
available electronic and optical components. The con- 

20 trol is principally edge-based and the core modules 34 
operate independently so that if one core module fails, 
the balance of the switch continues to operate and traffic 
is rerouted through the remaining available core mod- 
ules. Normal failover techniques well known in the art 

25 may be used to ensure continuous operation in the 
event of component failure. 

[0049] The embodiments of the invention described 
above are intended to be exemplary only. The scope of 
the invention is therefore intended to be limited solely 
30 by the scope of the appended claims. 

Claims 

35 1 . A high capacity packet switch comprising a plurality 
of core modules, each of the core modules (26) op- 
erating in a circuit switching mode, a plurality of 
edge modules (22, 24) connected to subtending 
packet sources and subtending packet sinks, each 

^0 of the edge modules operating in a data packet 
switching mode, CHARACTERIZED by: 

the core modules switch the data packets be- 
tween the edge modules using wavelength division 
multiplexing (WDM) and time division multiplexing 

45 (TDM). 

2. A high capacity packet switch as claimed in claim 1 
wherein each core module (26) is a space switch. 

so 3. a high capacity packet switch as claimed in claim 2 
wherein each core module (26) is a single-stage 
electronic rotator switch. 

4. A high capacity packet switch as claimed in any pre- 
55 ceding claim wherein: 

each edge module (22,24) has a plurality of in- 
gress ports, each of the ingress ports having an 
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associated ingress queue; and 
an ingress scheduler sorts packets arriving in 
the ingress queues from the subtending packet 
sources, the sort being by egress edge module 
(24) from which the respective packets are to 5 
egress from the high capacity packet switch 
(20) for delivery to the subtending packet sinks. 

5. A high capacity packet switch as claimed in claim 4 
wherein the ingress scheduler periodically deter- 10 
mines a number of packets waiting in the ingress 
queues for each respective egress edge module 
(22,24) and sends a capacity request vector to each 

of the controllers (22) of the core modules (34), the 
capacity request vector sent to a given controller re- *5 
lating only to a group of channels that connect the 
edge module to the given core module. 

6. A high capacity packet switch as claimed in claims 

4 or 5 wherein each ingress edge module (22) main- 20 
tains a vector of pointers (58) to the packets sorted 
by egress edge module (24) and a scheduling ma- 
trix (54) that provides a port number (56) for each 
slot in which a data block can be transferred, the 
scheduling matrix being arranged in the same 25 
egress edge module order so that the scheduling 
matrix and the pointers are logically aligned; and, 
when a non-blank entry in the scheduling matrix in- 
dicates an egress port through which a data block 
can be transferred, a corresponding pointer in the 30 
vector of pointers is used to locate a starting point 
for the data block in the packets waiting in the in- 
gress queues. 

7. A high capacity packet switch as claimed in any pre- 35 
ceding claim wherein the core modules (34) and the 
edge modules (22,24) are spatially distributed. 

8. A high capacity packet switch as claimed in claim 7 
wherein one edge module (22) is co-located with 40 
each core module (34) , and the edge module serves 

as a controller for the core module. 

9. A high capacity packet switch as claimed in claims 

7 or 8 wherein each edge module (22,24) has M 45 
reconfiguration timing circuits, where M is the 
number of core modules (34), each of the reconfig- 
uration timing circuits being time-coordinated with 
a time counter in the respective edge modules that 
serve as processors for the core modules, to coor- so 
dinate data transfer from the ingress edge modules 
when the core modules are reconfigured to change 
channel connectivity. 

10. A high capacity packet switch as claimed in any pre- ss 
ceding claim wherein each edge module (24,34) is 
connected to each core module by at least one com- 
munications link (25). 



11. A high capacity packet switch as claimed in claim 
1 0 wherein each core module (34) comprises a plu- 
rality of single-stage rotator switches (26), each ro- 
tator switch having a number of input ports collec- 
tively adapted to accommodate a number of chan- 
nels equal to the number of ingress edge modules 
(22,24) and a number of oulpul ports collectively 
adapted to accommodate a number of channels 
equal to the number of egress edge modules, and 
each edge module has at least one channel to each 
of the rotator switches. 

12. A high capacity distributed packet switch as claimed 
in any preceding claim wherein the ingress edge 
module (22) and the egress edge modules (24) 
comprise integrated units of one ingress edge mod- 
ule and one egress edge module each. 

13. A method of switching payload data packets 
through a distributed data packet switch (20). com- 
prising steps of receiving a payload data packet 
from a subtending source at an ingress edge mod- 
ule (22) of the distributed data packets switch, de- 
termining an identity of an egress edge module (24) 
from which the data packet should egress from the 
distributed data packet switch, arranging the data 
packet in a sorted order with other data packets re- 
ceived so that the data packets are arranged in a 
sorted order corresponding to the identity of the 
edge module from which the data packet should 
egress from the distributed data packet switch, 
transferring the sorted data packets in fixed-length 
data blocks to a core module (34) of the distributed 
data packet switch, switching the fixed-length data 
blocks through the core module to the egress mod- 
ule, reconstructing the data packet at the egress 
edge module, and transferring the data packet from 
the egress module to a subtending sink CHARAC- 
TERIZED by: 

using wave division multiplexing (WDM) and 
time division multiplexing (TDM) to provide intra- 
switch data paths of a fine granularity to reduce a 
requirement for tandem switching in the distributed 
data packet switch. 

14. A method as claimed in claim 1 3 wherein an ingress 
scheduler periodically determines a number of 
packets waiting in ingress queues for each respec- 
tive egress edge module (22) and sends a capacity 
request vector to each of the controllers (22) of the 
core modules (34), the capacity request vector sent 
to a given controller relating only to a group of chan- 
nels that connect the edge module to the given core 
module. 

15. A method as claimed in claim 14 wherein each in- 
gress edge module (22) maintains a vector of point- 
ers to the packets sorted by egress edge module 
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(24) and a scheduling matrix (54) that provides a 
port number for each slot in which a data block can 
be transferred, the scheduling matrix being ar- 
ranged in the same egress edge module order so 
that the scheduling matrix and the pointers are log- 5 
ically aligned; and, when a non-blank entry in the 
scheduling matrix indicates an egress port through 
which a data block can be transferred, a corre- 
sponding pointer in the vector of pointers is used to 
locate a starting point for the data block in the pack- 10 
ets waiting in the ingress queues. 

16. The method as claimed in claim 1 5 wherein an array 
(58) stores pointers to packets sorted by egress 
edge module, and the pointers are dynamically up- is 
dated each time a data block is transferred from the 
ingress queues to an egress channel so that recon- 
figurations of the core modules (30) can be accom- 
plished without queue flushing. 

20 

17. The method as claimed in claims 15 or 16 wherein 
two scheduling matrices (54). one in current use 
and one in update mode, are maintained at each 
ingress module (22) . 

25 

18. The method as claimed in claim 17 wherein each 
time a core reconfiguration occurs, a scheduling 
matrix (54) in use is swapped for a current sched- 
uling matrix. 

30 

19. The method as claimed in claim 17 or 1 8 wherein 
~an unused copy of the scheduling matrix (54) is 

available for update after the core reconfiguration. 

20. The method as claimed in claim 1 9 wherein rows in 35 
the matrix are executed sequentially, one per time 
slot, until a next core module reconfiguration oc- 
curs; and, after core module (34) reconfiguration, 
processing continues at a next time slot. 
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(54) High-capacity WDM-TDM packet switch 

(57) A self-configuring distributed packet switch(20) 
that operates in wavelength division multiplexed (WDM) 
and time division multiplexed (TDM) modes is de- 
scribed. The switch comprises a distributed channel 
switching core (26) , that includes core modules (34) 
which are respectively connected by a plurality of chan- 
nels to a plurality of high-capacity packet switch edge 
modules (22,24). Each core module operates independ- 
ently to schedule paths between edge modules, and 



reconfigures the paths in response to dynamic changes 
in data traffic loads reported by the edge modules. 
Reconfiguration timing between the packet switch mod- 
ules and the channel switch core modules is performed 
to keep reconfiguration guard time minimized. The ad- 
vantage is a high-capacity, load-adaptive, self-configur- 
ing switch that can be distributed to serve a large geo- 
graphical area and can be scaled to hundreds of Tera 
bits per second to support applications that require very 
high bandwidth and a guaranteed quality of service. 
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